Automatic Dictionary Construction and Identification of Parallel Text Pairs

When creating dictionaries for use in for example cross-language search engines, parallel or comparable text pairs are needed. Multilingual web sites may contain parallel texts but these can be difficult to detect. For instance, a multilingual website, Hallå Norden, contains information in five languages; Swedish, Danish, Norwegian, Icelandic and Finnish. Working with these texts we discovered two main problems: the parallel corpus was very sparse, containing on average less than 80.000 words per language pair (in the final version of the corpora), and it was difficult to automatically detect parallel text pairs. We discovered that, on average, around 55 percent of the texts were not parallel. Creating dictionaries with the word aligner Uplug gave on average 213 dictionary entries. Despite the corpus sparseness the results were surprisingly good compared to other experiments with larger corpora. Following this work, we made two sets of experiments on automatic identification of parallel text pairs. The first experiment utilized the frequency distribution of word initial letters in order to map a text in one language to a corresponding text in another in the JRCAcquis corpus (European Council legal texts). Using English and Swedish as language pair, and running a ten-fold random pairing, the algorithm made 87 percent correct matches (baseline-random 50 percent). Attempting to map the correct text among nine randomly chosen false matches and one true yielded a success rate of 68 percent (baseline-random 10